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Executive  Summary 


Introduction.  The  Air  Force  Personnel  Center’s  Force  Management  Liaison  Office  (AFPC/DPSF)  is 
frequently  called  upon  to  respond  to  human  capital  management  issues  that  periodically  arise  among 
stakeholders  of  the  Air  Force.  DPSF  directly  supports  the  Force  Management  Division  (HQ  AF/A1PF)  of 
the  Air  Staffs  Manpower  and  Personnel  Directorate.  In  order  to  support  AF/A1PF  in  a  timely  manner, 
DSYX  maintains  a  Human  Resources  Research  Databank  (HRRD),  a  repository  of  data  that  enables  a 
wide  range  of  research  questions  to  be  addressed  relatively  quickly.  The  Air  Force  Occupational  Analysis 
Program  of  the  Air  Force  Occupational  Measurement  Squadron  (AFOMS)  is  one  valuable  source  of 
information  for  the  HRRD,  and  in  2007,  DPSF  stated  a  requirement  for  a  plan  to  facilitate  the  systematic 
flow  of  completed  Occupational  Analysis  Reports  (OAR)  from  AFOMS  to  the  HRRD. 

Methodology.  The  approach  for  developing  the  Transition  Plan  was  three-phased.  In  the  first  phase, 
the  current  AFOMS  processes  for  gathering  data  and  developing  an  OAR  were  examined.  AFOMS  uses 
survey  methods  to  gather  task  performance  data  from  Air  Force  job  incumbents  and  task  factor  rating  data 
from  experienced  subject  matter  experts.  A  variety  of  analytic  tools  are  then  applied  to  interpret  the 
implications  of  the  data  for  issues  such  as  currency  of  training  standards,  classification  decisions, 
personnel  utilization  policies,  and  other  Air  Force  personnel  programs.  In  the  second  phase  of  this 
project,  the  tools  expected  to  be  useful  to  the  HRRD  were  examined  to  identify  data  requirements  which 
can  be  satisfied  by  AFOMS’s  methods.  In  the  final  phase,  an  analysis  was  conducted  to  identify  gaps 
between  data  available  from  the  OAR  process  and  those  data  needed  by  the  HRRD  tools.  The  third  phase 
closes  with  recommendations  on  how  to  resolve  gaps  between  available  and  needed  data. 

Phase  I.  The  process  for  developing  and  conducting  occupational  surveys  have  remained  relatively 
stable  for  many  years,  although  the  methodology  has  regularly  been  automated  and  streamlined  as  new 
tools  and  techniques  emerged.  The  most  substantial  recent  change  has  been  a  shift  in  how  the  data  are 
handled  after  collection  is  completed.  Data  are  now  converted  into  relational  tables  within  the  ORACLE 
database  system  as  opposed  to  the  previous  hierarchical  OA  study  approach,  with  ASCII  files  containing 
individual  records  for  each  respondent  within  a  given  study.  Despite  the  shift  to  a  relational  database 
format,  data  elements  remain  in  place  to  perform  basic  analyses  as  before;  and  it  should  be  possible  to 
reassemble  the  database  output  for  a  given  project  in  a  form  resembling  the  output  of  the  previous 
approach.  AFOMS  has  changed  analysis  software,  moving  from  the  Air  Force-developed  legacy 
Comprehensive  Occupational  Data  Analysis  Programs  (CODAP)  to  the  commercially  available  Statistical 
Package  for  the  Social  Sciences  (SPSS);  the  newer  software  is  reported  to  address  the  same  data-analytic 
functions  as  did  the  legacy  software,  however,  if  head-to-head  comparisons  have  been  made  to 
demonstrate  exact  equivalence  of  the  outputs,  this  author  has  not  seen  the  comparisons  published  or 
reported.  Some  analytic  procedures  are  no  longer  being  performed,  such  as  the  hierarchical  clustering  by 
task.  Other  procedures,  such  as  hierarchical  clustering  by  case,  are  being  performed  on  limited  numbers 
of  reports.  The  format  of  the  OAR  has  substantially  changed,  moving  from  a  Microsoft  Word™  report 
format  to  an  EXCEL™  report  style  with  tables  and  graphics  embedded  in  a  report  which  thoroughly 
presents  analyses  in  ways  that  address  current  users’  issues;  however,  most  of  the  critical  analytic  views 
within  the  OAR  remain  from  legacy  reports.  Interestingly,  current  reports  address  emerging  issues,  such 
as  differential  analyses  of  deployed  versus  home  station  work  responsibilities  and  distinctions  among 
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active  duty,  Air  National  Guard  and  Air  Force  Reserve  incumbents,  reflecting  a  responsible  approach  to 
performing  occupational  analysis  within  a  dynamically  changing  Air  Force. 

Phase  II.  AFPC/DPSF  identified  a  number  of  tools  for  addressing  questions  which  arise  as  force 
management  and  operational  tempo  issues  intersect.  They  recognize  the  potential  for  developing  new 
tools  should  requirements  arise,  to  provide  timely  yet  responsible  data-grounded  analyses  and 
recommendations  to  those  recurring  questions.  Key  among  those  tools  are: 

•  Training  Impacts  Decision  System  (TIDES), 

•  Training  Efficiency  and  Effectiveness  Methodology  (TEEM), 

•  Simulation  Utility  Analysis  System  (SUMS), 

•  Acquisition  Information  Retrieval  and  Simulation  (ACQUIRES),  and 

•  Job  Structuring  Technology  (JST). 


In  addition,  the  Benchmarked  Occupational  Learning  Difficulty  (BOLD)  technique,  developed  jointly  by 
the  Air  Force  Research  Lab  and  AFOMS,  has  potential  for  supporting  cross-occupation  analyses  not 
generally  supported  by  AFOMS’S  routine  analytic  practices.  The  data  requirements  documents  for  these 
tools  were  examined  to  determine  which  of  the  required  data  elements  could  be  supplied  by  the  current 
AFOMS  products  and  relational  database. 

Phase  III.  In  analyzing  the  gap  between  current  AFOMS/OA  outputs  and  those  requirements  of  the 
HRRD  tools,  most  of  the  needed  data  are  available;  however,  some  data  types  have  been  discontinued  or 
modified.  Task  factor  data  were  routinely  collected  on  all  tasks  within  a  given  survey  project;  but  in  an 
effort  to  be  responsive  to  their  primary  customers,  AFOMS/OA  modified  the  strategy  for  collecting  Task 
Learning  Difficulty  in  most  surveys,  restricting  data  collection  primarily  to  technical  tasks,  omitting  the 
supervisory,  planning  and  evaluating,  and  some  administrative  tasks.  Further,  while  the  ratings  are 
presented,  there  is  scant  evidence  the  data  have  been  evaluated  for  inter-rater  reliability  prior  to 
publishing.  The  TEEM  tool  is  somewhat  impacted  by  the  reduction  in  availability  of  some  Task  Learning 
Difficulty  data;  however,  the  impact  is  not  insurmountable.  One  tool,  TIDES,  was  founded  on  two 
principle  sources  of  occupational  analysis  data;  the  task  module  and  job  structure.  By  replacing 
hierarchical  case  clustering  in  some  studies  with  the  practice  of  trusting  job  incumbents  to  select  their  job 
title  from  a  list  of  expected  titles,  the  question  of  validity  of  the  job  structure  arises.  Reduced  validity  of 
the  job  structure  can  impact  the  effectiveness  of  TIDES.  Task  clustering  forms  task  modules,  the  building 
blocks  of  jobs  within  TIDES.  Therefore,  the  elimination  of  task  clustering  risks  TIDES  becoming  less 
effective  in  modeling  the  utilization  patterns  and  training  requirements  for  Air  Force  occupations. 
Questions  about  validity  of  job  structures  also  risk  impacting  other  tools,  such  as  SUMS,  ACQUIRES, 
and  JST.  Addressing  these  data  shortfalls  will  not  be  a  simple  matter;  however,  it  would  be  worthwhile 
for  DPSF  to  have  the  ability  to  resurvey  occupations  to  gather  supplemental  data  in  a  timely  fashion  not 
captured  by  AFOMS’Ss  primary  surveys.  Given  the  available  AFOMS/OA  products,  that  should  be 
feasible  on  a  project  by  project  basis  from  within  the  HRRD. 
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Transition  Plan  for  Moving  AFOMS  Occupational  Analysis  Report  Contents 


to 

AFPC’s  Human  Resources  Research  Databank 


Randall  C.  Agee 

Operational  Technologies  Corporation 


INTRODUCTION 

As  Air  Force  organizational  structures  have  changed  over  the  years,  responsibility  for  answering  human 
resource  policy  issues  has  more  clearly  rested  with  Headquarters  Air  Force,  Directorate  of  Manpower  and 
Personnel  (HQ  AF/A1).  As  the  Field  Operating  Agency  of  AF/A1,  the  Force  Management  Liaison  Office 
(DPSF)  of  the  Air  Force  Personnel  Center  (AFPC)  (see  12  FTW,  2008)  is  tasked  with  developing  and 
sustaining  a  data  repository  to  support  AF/A1  in  addressing  the  short -notice  human  resources  policy 
issues.  That  repository,  the  Human  Resources  Research  Databank  (HRRD),  consists  of  collections  of 
research  and  operational  data  sources  that  enable  AFPC/DPSF  and  AF/A1  to  respond  in  a  timely  fashion 
to  inquiries  regarding  the  likely  effects  of  various  policy  options  facing  Air  Force  stakeholders.  This 
Databank  houses  both  current  and  historical  data  for  use  in  addressing  both  longitudinal  and  cross¬ 
specialty  questions  relating  to  personnel  recruiting,  selection,  training,  employment,  and  promotion 
(Christal,  1974).  The  data  are  expected  to  be  useful  in  developing  screening  criteria  for  selecting  new 
accessions,  making  effective  personnel  classification  and  assignment  determinations,  and  designing 
Promotion  Fitness  Exams  (PFEs)  and  Specialty  Knowledge  Tests  (SKTs)  to  support  the  Weighted 
Aimian  Promotion  Testing  System  (WAPS).  It  may  also  enable  answers  to  questions  of  the  viability  of 
options  for  AFS  restructuring  initiatives,  the  utility  of  job  perfomiance  measures  in  managing  the  force, 
and  application  of  personality  variables  in  ensuring  effective  and  efficient  use  of  human  capital  in 
assignments  (person-job-match).  Some  of  the  tools  at  the  HRRD’s  disposal  include  the  Training  Impacts 
Decision  System  (TIDES),  Training  Effectiveness  and  Efficiency  Methodology  (TEEM),  Simulation 
Utility  Analysis  System  (SUMS),  Acquisition  Information  Retrieval  and  Simulation  (ACQUIRES),  and 
Job  Structuring  Technology  (JST),  as  well  as  the  selection  instruments  such  as  the  Armed  Forces 
Vocational  Aptitude  Test  Battery  (ASVAB)  and  the  Air  Force  Officer  Qualifying  Test  (AFOQT). 

The  Air  Force  Occupational  Analysis  Program  of  the  Air  Force  Occupational  Measurement  Squadron 
(AFOMS)  is  one  of  several  valuable  sources  of  information  for  the  HRRD.  DPSF  has  a  requirement  for  a 
plan  to  facilitate  the  systematic  flow  of  completed  Occupational  Analysis  Reports  (OAR)  from  AFOMS 
to  the  HRRD.  This  report  presents  that  plan. 
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METHODOLOGY 


The  approach  for  developing  this  Transition  Plan  was  discussed  at  a  contract  kick-off  meeting  held 
November  29th,  2007  at  HQ  Air  Force  Recruiting  Service,  Randolph  AFB  TX.  The  initial  intent  was  to 
have  the  plan  developed  in  three-phases.  In  the  first  phase,  the  current  AFOMS  processes  for  gathering 
data  and  developing  an  OAR  were  to  be  examined,  with  attention  paid  to  the  processes  used  to  gather  data 
from  Air  Force  job  incumbents  and  subject  matter  experts  and  to  convert  that  data  into  formats  needed  for 
currently  used  analytic  tools  and  reporting  practices.  Information  gathering  was  to  be  performed  by 
interviewing  key  AFOMS  staff  members,  especially  those  responsible  for  constructing  occupational 
surveys,  those  responsible  for  converting  the  surveys  into  automated  form  for  online  data  collection,  those 
responsible  for  converting  the  resultant  raw  data  into  data  files  suitable  for  the  analysis  software,  and 
those  responsible  for  analyzing  results,  preparing  finished  products,  and  working  with  customers  to  apply 
the  analysis  results  to  ongoing  Air  Force  applications. 

In  the  second  phase,  the  tools  expected  to  be  useful  to  the  HRRD  were  to  be  examined  to  identify  data 
requirements  which  can  be  satisfied  by  AFOMS’  methods.  Conversations  with  AFPC/DPSF,  HQ  AF/A1, 
and  other  interested  stakeholders  suggested  a  modest  number  of  tools  that  should  be  considered.  The  plan 
was  to  examine  published  research  and  extant  documentation  for  the  specified  tools  to  identify  key  data 
requirements  that  may  be  available  through  AFOMS’  products. 

In  the  final  phase,  an  analysis  was  conducted  to  identify  gaps  between  data  available  from  the  OAR 
process  and  data  needed  by  the  HRRD  tools.  The  third  phase  was  expected  to  close  with 
recommendations  on  how  to  resolve  gaps  between  available  and  needed  data. 

It  was  necessary  to  modify  this  methodology  several  times  during  the  course  of  the  project,  due  to 
unanticipated  limitations  in  availability  of  AFOMS  personnel.  At  the  time  the  project  kicked  off,  there 
was  an  announcement  that  the  Air  Force  Management  Agency  (AFMA)  was  initiating  a  possible  A-76 
Study  which  would  evaluate  the  work  being  done  by  AFOMS  to  determine  if  it  is  inherently 
governmental  and  if  not,  consider  whether  it  should  be  outsourced  (OMB,  20003).  Due  to  the  sensitivity 
of  the  study,  requests  for  needed  interviews  were  postponed,  and  then  later,  declined  as  the  decision 
process  worked  its  way  through  stakeholder  coordination  channels.  In  March  2008,  permission  was 
received  to  conduct  some  interviews.  Two  were  accomplished,  with  staff  members  of  the  Occupational 
Analysis  Section  (AFOMS/OAO)  and  the  Inventory  Development  Section  (AFOMS/OAV).  One  further 
interview  was  scheduled,  (with  a  representative  of  the  Resource  Management  section  (AFOMS/RM); 
however,  that  interview  was  canceled  after  (1)  the  AFOMS  senior  staff  briefed  AFPC/DPSF  and  AF/A1 
personnel  on  current  methodology,  and  (2)  this  author  had  a  telephone  dialogue  with  the  RM  chief  who 
answered  many  of  the  outstanding  questions.  The  information  gathering  for  Phase  I  was  further 
complicated  by  a  number  of  staff  changes  at  AFOMS  in  early  2008,  resulting  in  uncertainties  about  the 
future  direction  of  AFOMS.  That  future  direction  was  finally  resolved  when  the  decision  was  reached 
that  the  A-76  Study  would  not  be  conducted,  but  30+  military  positions  within  AFOMS  would  be 
reallocated  to  the  Air  Force  and  the  smaller-sized  Occupational  Measurement  Squadron  would  be  staffed 
completely  by  civil  service  personnel. 
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RESULTS 


Phase  I.  Rumors  of  dramatic  change  in  the  OAR  process  are  somewhat  overstated.  The  process  for 
developing  and  conducting  occupational  surveys  (Christal,  1974)  has  remained  relatively  stable  since  the 
program  was  founded,  although  many  mundane  and  tedious  aspects  have  been  automated  and 
streamlined,  and  presentation  of  the  survey  instrument  has  been  realigned  to  focus  first  on  the  more 
technical  aspects  of  each  occupation’s  work  as  opposed  to  starting  with  leadership,  supervisory,  training, 
and  administrative  tasks. 

Conceptually,  data  processing  is  currently  handled  in  a  different  way  than  it  was  in  earlier  years  of  the 
program.  Raw  survey  data  are  now  converted  into  relational  tables  within  the  ORACLE  database  system 
as  opposed  to  the  previous  hierarchical  OA  study  approach,  with  ASCII  files  containing  individual 
records  for  each  respondent  within  a  given  study.  Despite  the  shift  to  a  relational  database  format,  data 
elements  remain  in  place  to  perform  basic  analyses  as  before;  and  it  should  be  possible  to  reassemble  the 
database  output  in  a  form  resembling  the  form  in  which  the  data  were  collected. 

AFOMS  has  changed  analysis  software,  moving  from  the  legacy  Comprehensive  Occupational  Data 
Analysis  Programs  (CODAP),  a  suite  of  tools  developed  by  the  Air  Force  Research  Laboratory  (Christal, 
1974,  and  Christal  &  Weissmuller,  1988)  to  the  commercially  available  Statistical  Package  for  the  Social 
Sciences  (SPSS).  The  newer  software  is  advertised  to  address  the  same  data-analytic  functions  as  did  the 
legacy  software,  but  has  the  advantage  of  being  more  intuitive  and  having  a  more  user-friendly  graphic 
interface;  however,  if  head-to-head  comparisons  have  been  made  to  demonstrate  exact  equivalence  of  the 
outputs,  this  author  has  not  seen  the  comparisons  published  or  reported.  Some  analytic  procedures  are  not 
being  performed  any  more,  such  as  the  hierarchical  clustering  by  task,  while  other  procedures,  such  as 
hierarchical  clustering  by  case,  are  being  performed  on  limited  numbers  of  reports. 

The  format  of  the  OAR  has  substantially  changed,  moving  from  a  Microsoft  Word™  report  format  to  an 
EXCEL™  report  style  with  tables  and  graphics  embedded  in  a  report  which  thoroughly  presents  analyses 
in  ways  that  address  current  users’  issues;  however,  most  of  the  critical  analytic  views  within  the  OAR 
remain  from  legacy  reports,  such  as  specialty  job  structures,  analysis  of  skill  level  differences,  training 
analysis  (to  include  first  job  tasks,  first  enlistment  tasks,  comparison  to  specialty  training  standards,  and 
task  factor  analyses),  differential  analysis  of  major  command  (MAJCOM)  requirements,  and  job 
satisfaction/retention  analyses.  Interestingly,  current  reports  address  emerging  issues,  such  as  differential 
analyses  of  deployed  versus  home  station  work  responsibilities  and  differential  analyses  of  distinctions 
among  active  duty,  Air  National  Guard  and  Air  Force  Reserve  incumbents,  reflecting  a  responsible 
approach  to  performing  occupational  analysis  within  a  dynamically  changing  Air  Force. 

The  following  paragraphs  describe  conversations  held  with  AFOMS  personnel  during  the  month  of  April. 
They  are  not  in  chronological  order.  The  first  two  conversations,  discussing  the  OA  process  are  presented 
first,  followed  by  results  of  a  telephone  conversation  on  how  data  are  processed  in  the  current  approach. 
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Discussion  with  AFOMS/OAV: 


The  author  met  with  three  staff  members  of  the  combined  Inventory  Development  and  Data  Collection 
section,  AFOMS/OAV,  April  30th  (McDaniel,  Ostrander,  &  Cluxton,  2008)  and  received  excellent 
cooperation  and  patience.  We  discussed  front-end  procedures  of  the  OA  process.  One  interviewee 
headed  the  section  constructing  job  inventories.  Another  interviewee  was  long-time  technical  expert  in 
automating  surveys  for  field  administration,  and  the  third  interviewee  had  been  selected  to  assume  the 
role  of  automating  surveys  for  administration.  We  discussed  their  process  from  initiating  new  surveys  to 
retrieving  recently  archived  inventories,  validating  and  updating  previous  iterations  of  the  job  inventories, 
readying  new  job  inventories  for  administration,  soliciting  and  managing  data  collection,  performing 
quality  assurance  of  incoming  data,  and  submitting  the  resulting  data  files  to  AFOMS/RM  for  further 
processing. 

In  current  procedures,  when  a  survey  is  completed,  the  raw  data  file,  an  ACCESS  file  (*.mdb)  is  cleaned 
to  remove  grossly  flawed  cases  (e.g.,  respondents  who  began  a  survey  in  error  and  realized  the  survey  was 
not  for  their  Air  Force  Specialty  [AFS],  or  respondents  who  quit  the  survey  early  and  gave  an  obviously 
incomplete  record).  Output  from  that  process,  is  analogous  to  our  old  raw  “input  standard”  file.  Under 
current  policy,  a  copy  of  the  cleaned  data  file  is  retained  by  AFOMS/OAV  and  a  copy  is  submitted  to 
AFOMS/RM  for  entry  into  ORACLE.  During  the  change-over  from  using  CODAP  for  analysis  to  using 
SPSS,  there  was  a  gap  of  approximately  two  years,  when  cleaned  files  were  not  retained.  As  a  result, 
only  the  RM  section  will  have  historical  versions  of  the  “input  standard  analog”  files  during  that  period. 

Discussion  with  AFOMS/OAO: 


The  author  met  with  two  staff  members  of  AFOMS/OAO  April  16th  (Dubois  and  Luster,  2008)  and 
received  excellent  cooperation  and  patience.  They  explained  the  current  AFOMS/OA  organizational 
structure  and  described  the  current  process  used  in  conducting  occupational  analysis.  The  program 
continues  to  follow  historical  precedents  for  the  OA  process  as  it  has  been  practiced  for  over  25  years 
(Christal,  1974),  but  has  integrated  new  technical  features  to  accelerate  and  streamline  processes.  Despite 
plans  to  reengineer  the  program,  there  are  still  designated  sections  of  staff  devoted  to  developing  job 
inventories,  collecting  the  survey  data,  and  performing  analysis,  preparing  reports  and  delivering  results 
to  the  training  community  at  Utilization  and  Training  Workshops.  This  organizational  structure  seems  to 
satisfy  both  the  volume  and  quality  concerns  as  the  program  strives  to  satisfy  key  customers’  needs.  The 
definition  of  “customer”  has  evolved  somewhat  over  the  years,  as  the  Air  Force  adjusted  to  technology 
changes,  evolving  mission  requirements  and  budgetary  realities.  Today,  the  OA  Program  views  their 
primary  customer  as  the  training  community.  While  that  community  has  always  been  a  major  consumer 
of  OA,  other  customers,  such  as  the  recruiting,  personnel  classification,  and  promotion  testing 
communities  seem  to  have  become  of  less  pressing  importance. 

The  OA  program  converted  their  reports  from  Microsoft  Word™  format  into  an  EXCEL  Workbook™ 
format  which  retains  and  integrates  much  of  the  narrative  from  former  reports  with  the  many  tables  of 
data  viewing  survey  results  from  a  variety  of  viewpoints.  One  key  analytic  procedure,  the  job  structure 
analysis  procedure  changed  substantially  over  the  past  half-dozen  years.  For  many  projects,  the  formal 
job  typing  techniques  are  no  longer  used,  but  have  been  replaced  with  a  strategy  of  depending  on  job 
incumbents  to  sort  themselves  by  selecting  their  job  title  from  a  list  provided  in  the 
Biographical/Background  section  of  job  inventories.  While  this  strategy  has  some  appeal,  it  is  fraught 
with  risks.  Depending  on  the  descriptive  quality  of  job  titles,  the  complexity  of  the  specialty,  and  the 
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degree  of  specialization  within,  incumbent-defined  job-type  groupings  contain  a  great  deal  of  variance 
and  do  not  capture  the  nuances  of  job-type  groupings  derived  from  the  job  clustering  approach.  For 
instance,  it  is  possible  for  incumbents  to  inflate  the  importance  of  their  jobs  by  selecting  a  title  that  should 
require  special  training,  experience  or  other  qualifications.  In  some  AFSs,  technological  standardization 
has  driven  incumbents  into  very  homogeneous  work  patterns,  and  most  members  respond  not  to 
descriptive  job  titles,  but  to  the  official  title  of  the  specialty  to  which  they  belong  (e.g.,  “Fm  a  Pavements 
and  Construction  Equipment  Specialist,  not  a  Florizontal  Fleavy  Equipment  Operator  or  a  Tool  Crib 
Manager”. . .).  Flowever,  there  are  still  some  projects  where  the  formal  Job  Typing  methodology  is  still  in 
use,  applying  the  hierarchical  case  clustering  methodology  originally  endorsed  by  Christal  (1974)  as  a 
way  of  revealing  the  natural  organization  of  work  into  “jobs”  and  groups  of  jobs.  A  question  remains, 
however,  as  to  whether  the  results  of  Job  Typing  using  the  SPSS  analysis  package  is  equivalent  to  that 
provided  by  the  legacy  system,  since  to  this  author’s  knowledge,  no  head-to-head  comparisons  have  been 
reported.  SPSS  contains  at  least  two  types  of  hierarchical  clustering,  the  Ward  Method  and  the  D2 
Minimizing  Sum  of  Squares.  The  former  was  the  method  used  in  prior  years  analysis  and  was  adopted 
because  it  presents  a  more  accurate  picture  of  occupations  when  large  numbers  of  variables  (e.g.,  tasks) 
are  involved,  since  it  forces  membership,  at  each  level,  into  one  group  or  another.  The  latter  method, 
sometimes  called  the  Q-factor  hybrid  method,  allows  for  membership  in  multiple  groups,  and  thus, 
produces  a  less-clearly  defined  job  structure  (Levine,  2003). 

The  OA  interviewees  described  the  data  flow  from  the  time  a  survey  project  data  set  arrives  in  their 
section  through  their  clean-up  and  analysis  processes.  They  indicated  that  AFOMS’  RM  section  now 
centrally  processes  the  data  collected  and  imports  it  into  the  ORACLE  database.  The  output  is 
subsequently  formatted  for  analysts  in  a  product  called  PREPE,  akin  to  the  refined  “Input  Standard”  file, 
and  representing  the  original  data  set  with  the  most  extreme  unqualified  cases  eliminated.  Analysts  use 
the  PREPE  data  (which  pulls  data  from  another  program  called  CAROMS,  the  mailing  list  generator)  to 
ensure  remaining  cases  fit  the  criteria  for  inclusion  in  the  analysis  (member  is  available  for  duty,  has 
correct  AFS,  has  at  least  6  months  on  the  job,  is  not  imminently  changing  permanent  stations  nor  retiring 
during  the  planned  survey  administration  period).  From  there,  the  data  is  input  into  SPSS  for  some 
analyses,  and  then  transferred  to  EXCEL  for  creation  of  standard  and  custom  tables. 

The  interviewees  provided  examples  of  several  projects,  showing  the  new  layout,  including  one  study 
where  the  traditional  job-typing  analysis  was  being  conducted.  Although  the  formats  are  new  and  fresh, 
the  essential  structure  remains.  In  one  report,  the  new  format  follows  the  traditional  research  layout  of 
providing  an  executive  overview,  describing  the  methodology  used,  describing  the  survey  sample,  with 
emphasis  on  demonstrating  its  representativeness  of  the  population  being  studied,  and  moves  into  a 
results  phase,  where  various  analyses  are  discussed  in  a  rational  order.  Included  within  the  analyses  are 
the  key  sections  of  interest  to  the  training  community  (analysis  of  initial  first  job  and  first  enlistment 
training  requirements,  evaluation  of  support  for  Specialty  Training  Standards).  There  was  some 
description  of  the  process  used  to  gather  task  factor  (task  learning  difficulty  and  training  emphasis) 
ratings,  but  this  author  didn’t  see  any  information  describing  the  sampling  data  and  representativeness  of 
the  Subject  Matter  Experts,  nor  any  measure  of  their  degree  of  agreement  in  providing  task  factor  data 
(Mitchell,  1994),  although  the  ratings  summary  scores  were  included. 

In  another  example,  the  task  factor  methodology  was  even  less  clearly  presented.  The  reports  conclude 
with  a  variety  of  analyses  regarding,  differences  in  employment  by  major  commands,  employment  at 
home  station  verses  deployed,  total  force  issues,  and  job  satisfaction  and  retention  analyses. 
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The  interviewees  reported  that,  while  the  OA  program  is  still  willing  to  perform  officer  and  special 
management  studies,  most  of  their  efforts  are  focused  on  satisfying  requirements  for  collecting  and 
processing  data  to  address  training  issues  on  a  3 -year  cycle  covering  the  entire  spectrum  of  enlisted 
specialties.  In  large  measure,  this  is  due  to  a  resource  constraint. 

We  concluded  by  revisiting  discussions  of  interfaces  with  other  sections;  specifically,  work  done  by 
inventory  development  and  data  collection  sections  and  the  AFOMS/RM  interface.  As  online  surveys  are 
completed  by  job  incumbents,  data  are  stored  temporarily  in  ACCESS  databases,  then  cleaned  and 
imported  into  the  enteiprise  ORACLE  database.  The  AFOMS/RM  staff  process  those  data  and  make 
them  available  to  analysts,  who  work  off-line  to  construct  the  Occupational  Analysis  Report  (OAR), 
building  tables  and  illustrations,  analyzing  nuances  within  the  data,  writing  narratives  to  describe  method, 
interpret  results,  and  compose  descriptions  of  findings  with  the  potential  interest  to  users.  The  OAR  is 
very  training-oriented,  but  could  be  much  broader  in  impact  if  potential  users  framed  challenging 
questions  that  helped  direct  analysts’  attention  beyond  the  training  issues.  In  absence  of  regular  dialogue 
on  problems  other  than  training,  their  attention  is  directed  to  addressing  immediate  needs  of  the  training 
community. 

Discussion  with  AFOMS/RM: 

The  author  made  numerous  attempts  to  meet  during  the  month  of  April  with  the  Chief,  AFOMS/RM,  but 
due  to  leave  and  TDY  schedules  on  his  part,  we  were  unable  to  arrange  a  meeting.  Several  issues  were  of 
interest  in  having  the  meeting  with  the  AFOMS/RM.  From  previous  interviews,  it  appeared  the  survey 
data  collected  via  MTlSurv  were  being  processed  into  an  ACCESS  data  file  and  transferred  to  RM  for 
import  into  the  ORACLE  database  for  analysis.  Since  each  survey  layout  may  be  different,  it  is  important 
to  the  success  of  this  Transition  Plan  to  know  the  precise  file  names,  structures  and  procedures  to  interpret 
the  unique  survey  data  records  in  order  to  catalogue  each  survey  into  FIRRD.  Without  this  fundamental 
information,  AFPC/DPSF  is  left  having  to  reverse  engineer  every  step  of  the  analysis  process.  Also,  it  is 
important  to  the  success  of  this  Transition  Plan  to  have  both  the  “raw  data  file”  and  the  “processed  data 
file  from  the  final  report.”  The  needs  are  evident.  The  final  files  should  match  the  results  of  the  OA  and 
provide  support  and  explanation  for  report  contents.  The  “raw”  files,  on  the  other  hand,  enable  extended 
analysis  into  issues  associated  with  a  given  AFS  which  was  not  a  stated  requirement  when  survey  was 
initiated,  but  could  arise  after  the  OA  was  completed  and  archived.  In  addition,  the  “raw”  file  would 
provide  a  foundation  for  meta-analysis  across  multiple  surveys  on  comparative  analysis  issues  not 
addressed  in  the  original  AFOMS  mission. 

This  author  also  planned  to  discuss  with  the  RM  Chief  the  specific  SPSS  algorithm(s)  used  to  perform 
cluster-merger  analysis  in  those  cases  where  job  typing  is  done.  At  issue  is  that  during  their  reengineering 
of  the  OA  process,  RM  have  not  been  very  clear  that  the  current  algorithm  is  the  same  one  used  in  the 
previous  analysis  package,  and  raises  questions  of  equivalency  in  the  final  product.  If  any  head-to-head 
analyses  have  been  performed,  this  author  is  unaware  of  any  report  or  publication  showing  that 
equivalency. 

The  AFOMS/RM  and  this  author  did  schedule  a  meeting  for  May  5th.  Unfortunately,  the  May  5th 
meeting  was  rescheduled  two  times,  and  then  cancelled  by  a  telephone  call  the  morning  of  May  9th 
(AFOMS/RM,  2008).  In  cancelling  the  meeting,  the  RM  stated  that  the  AFOMS/CC  had  met  the 
previous  day  with  AFPC/DPSF  staff,  along  with  a  representative  of  HQ  AF/A1PF,  and  the  head  of  the 
AFS021  team  conducting  a  reengineering  review  of  the  OA  program.  The  Commander  stated  that 
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AFPC/DPSF  will  receive  “all  the  data  from  AFOMS  as  each  survey  is  completed.”  The  RM  Chief 
stressed  that  the  data  are  stored  by  ORACLE  in  a  relational  database  and  that  it  would  be  “millions  of 
lines  of  code”  for  each  survey.  As  we  discussed  the  future  directions  for  this  Transition  Plan,  the  RM 
Chief  suggested  that,  perhaps,  he  could  forward  a  sample  study  to  AFPC/DPSF  for  evaluation. 

On  May  15th,  AFPC/DPSF  forwarded  some  sample  data  from  AFOMS/RM  for  review  and  comment.  It 
appeared  the  ORACLE  output  of  “raw  data”  was  similar  to  the  data  used  to  process  the  recent  Similarity 
of  Skills  project;  the  AFOMS  data  appeared  to  have  been  converted  to  a  series  of  ACCESS  tables  for  the 
analysts  to  work  on  using  their  preferred  analysis  software  suite,  SPSS.  It  did  not,  however,  show  any 
resemblance  to  the  “individual  record”  format  that  is  output  from  MTlSurv.  AFPC/DPST  was  interested 
in  whether  the  ORACLE  output  could  be  worked  backward  to  reassemble  the  “individual  record”  format 
that  would  be  useful  for  input  into  TIDES,  TEEM,  SUMS  and  other  tools  envisioned  for  FIRRD.  They 
inquired  whether  the  MTlSurv  Utilities  program  could  do  such  reassembly.  In  its  current  configuration 
MTlSurv  Utilities  will  not  reverse  the  conversion  to  ORACLE  (Metrica,  2002),  but  it  could  be  modified, 
or  an  alternative  program  developed,  to  assemble  a  new  file  from  the  ORACLE  output  into  a  format 
equivalent  to  one  that  would  have  resulted  from  the  MTlSurv  output,  provided  key  information  about 
structures,  variable  names,  etc.,  of  the  original  survey  instruments  are  retained.  The  essential 
requirements  remain:  there  must  be  a  clear  definition  of  files  that  went  into  a  given  survey  at  the 
beginning  of  analysis.  With  the  raw  ORACLE  output  and  a  definition  of  the  records  that  were  used  to 
input  them,  a  relatively  easy  program  could  be  developed  to  reassemble  the  outputs  into  the  original 
records,  and  formatted  in  a  number  of  options,  including  the  very  basic  ASCII  “comma-separated 
variable”  layout,  and  from  there,  manipulated  by  a  number  of  analysis  tools. 

Phase  II.  AFPC/DPSF  and  AF/A1  recognize  that  there  are  persistent  questions  which  arise  as  force 
management  practices,  budgetary  realities,  and  operational  tempo  issues  intersect.  DPSF  has  identified  a 
number  of  tools  available,  and  recognizes  the  potential  for  developing  new  tools  should  the  requirements 
arise,  to  provide  timely  yet  responsible  data-grounded  analyses  and  recommendations  to  those  recurring 
questions.  Many  of  the  tools  grew  from  research  in  human  capital  management  that  was  funded, 
supervised,  and  in  many  cases,  performed  by  the  Air  Force  Research  Laboratory  and  its  predecessor 
organizations.  Key  among  those  tools  are  TIDES,  TEEM,  (SUMS,  (ACQUIRES,  and  JST.  In  addition, 
the  BOLD  technique  (Mitchell,  Tucker,  Fast,  Bennett,  &  Albert,  1997;  Mitchell,  1994),  developed  jointly 
by  the  Air  Force  Research  Lab  and  AFOMS,  has  potential  for  supporting  cross-occupation  analyses  not 
generally  supported  by  AFOMS’  usual  analytic  practices.  The  data  requirements  documents  for  these 
tools  were  examined  to  determine  which  of  the  required  data  elements  could  be  supplied  by  the  current 
AFOMS  products  and  relational  database. 


Training  Impacts  Decision  System  (TIDES)  Requirements  '. 

The  TIDES  is  a  technology  that  extends  traditional  occupational  analysis  by  creating  an  entity-based 
model  of  the  flow  of  personnel  through  an  Air  Force  Specialty  (AFS).  In  concept,  it  captures  a  snapshot 
of  the  entire  population  (or  at  least,  a  representative  sampling  of  members)  in  various  stages  of  career 
development.  It  views  an  AFS  in  terms  of  “career  cohorts”  that  enter  at  the  basic  training  point,  flow 
together  through  technical  training,  and  then  are  assigned  to  various  initial  jobs.  From  the  initial  jobs, 
they  subsequently  move  as  cohort  stages  are  crossed,  from  one  job  to  another.  At  some  stages,  they  pass 
through  professional  military  education  and  other  specialized  and  advanced  training  events.  From  one 

7 


stage  of  progression  to  another,  the  cohorts  shrink  with  observed  attrition.  TIDES  provides  cost  and 
capacity  estimates  based  on  a  number  of  standardized  values,  such  as  per  diem,  airline  rates,  and  salaries, 
as  well  as  some  nuanced  values,  such  as  the  length  of  technical  training  for  different  AFSs,  probabilities 
of  movement  from  a  lower-level  job  to  higher-level  jobs  (with  attendant  training  costs),  attrition  rates  at 
different  career  stages,  and  availability  of  resources  for  use  in  training.  The  resulting  patterns  of 
assignment  and  career  progression  flow  represent  an  “As-Is”  model  of  the  career  field.  Subsequently, 
policy  changes  (such  as  lengthening  “average  time  on  station”  or  “restructuring  particular  jobs”)  are 
imposed  on  the  “As-Is”  to  create  various  “To-Be”  scenarios,  and  differential  analysis  will  reveal  the 
advantages  and  challenges  of  making  those  changes  -  before  budgets,  individuals,  and  missions  are 
impacted.  A  number  of  data  elements  are  readily  available  (e.g.,  salary  rates,  pediem  rates  for  various 
locales,  air  fare  and  TDY  reimbursement  rates).  Other  data  are  uniquely  generated  within  the  TIDES 
methodology  (Stone,  Vasquez-Maury,  Fast,  and  Borden,  2005;  Mitchell,  et  al.,  1997;  Mitchell  and  Stone, 
1997;  Mitchell,  Bennett,  Wimpee,  Grimes,  and  Stone,  1995;  and  Chin,  Lamb,  Bennett  and  Vaughn, 

1992).  Those  unique  data  sets  include: 

A.  Job  and  Training  History  (J&TH)  Survey,  capturing  personal  history  of  job  assignments,  formal 
training  courses,  and  professional  military  education  of  a  large  sample  of  personnel  and  AFS; 

B.  Task  Module-to-Jobs  Map  (TM-Jobs  Map),  identifying  linkages  between  Job  Titles  and  TMs; 

C.  Resident  Course  Training  Times  Questionnaire  (TT),  collecting  high  fidelity  information  on  time 
dedicated  to  task  modules  (TM)  within  formal  training  courses  broken  out  by  hours  dedicated 
each  TM  in  classroom,  laboratory  and  self-study  as  well  as  hours  instructors  spent  preparing  to 
teach  each  knowledge  or  performance  requirement; 

D.  Training  Resource  Requirements  Questionnaire  (TRR),  identifying  training  resources  required  for 
each  course,  student  capacity  of  each  course,  and,  resource  capacities  available  to  each  course; 

E.  Task  Module  Learning  Curves  Questionnaire  (LC),  collecting  data  on  maximum  attainable 
proficiency  by  an  individual  using  specified  training  delivery  methods  (classroom,  laboratory, 
self-study  and  on-the-job-training,  or  UQT/OJT),  and  hours  required  to  attain  the  “go/no-go” 
level  of  proficiency;  and 

F.  Unit  Qualification  Training/On  the  Job  Training  Resource  Availability  Questionnaire  (TRA), 
estimating  capacity  of  exemplar  operational  units  to  conduct  UQT/OJT,  (specifically,  availability 
of  qualified  trainers  as  well  as  flow  of  trainees  through  UQT/OJT). 

AFOMS  data  can  significantly  support  TIDES  analysis.  There  are  two  key  elements  associated  with  the 
Job  Structure  analysis  that  are  integral  to  effective  TIDES  analyses.  First  is  quality  of  the  job  structure 
itself;  TIDES  models  the  jobs  that  make  up  the  AFS,  and  links  jobs  by  specifying  the  probability  of 
progression  from  any  one  job  to  another  in  the  next  higher  cohort.  To  construct  high-fidelity  TIDES 
models,  it  is  important  that  the  job  structure  is  as  close  to  the  real  world  as  possible.  The  second  element 
is  quality  of  Task  Modules  (TMs).  In  TIDES,  TMs  are  viewed  as  the  building  blocks  of  jobs.  In  contrast, 
OARs  view  the  tasks  themselves  as  the  building  blocks  of  jobs.  Thus,  to  perform  TIDES  successfully,  it 
is  necessary  to  group  tasks  according  to  some  taxonomy  that  works  to  reduce  the  number  of  building 
blocks  involved  to  a  manageable  level.  One  method  that  has  been  used  is  the  co-performance  clustering 
technique,  which  hierarchically  groups  tasks  with  other  tasks  that  have  a  high  probability  of  being 


performed  together.  This  technique  creates  groups  of  tasks  that  often  share  key  features  of  common  tacit 
and  explicit  knowledge,  common  tools,  and  common  processes.  It  is  not  perfect,  and  requires  analyst 
judgment  to  determine  when  task  modules  should  be  separated  and  when  they  should  be  joined,  but  in 
general,  it  reduces  the  numbers  of  items  in  the  TIDES  analysis  to  a  manageable  level,  and  enables  Subject 
Matter  Experts  (SME)  providing  other  data  (the  LCs,  the  TT,  TRR  and  TRA  data  in  particular)  to  make 
better  estimates  of  time  and  resources  required  to  achieve  proficiency  through  training.  Once  the  TMs  are 
created  using  co-performance,  it  is  a  fairly  easy  step  to  cross-compare  the  task  modules  with  the  jobs,  by 
cohort  group,  to  see  which  TMs  make  up  which  jobs,  and  whether  proficiency/competency  gained  in 
early  jobs  should  transfer  to  more  senior  jobs  as  the  individuals  progress  from  entry  through  career  stages. 
Creation  of  TMs  using  co-performance  is  a  fairly  easy  routine  to  run,  but  doesn’t  have  immediate  bearing 
on  AFOMS’  primary  customer,  so  it  has  been  discontinued.  An  alternate  approach  to  creating  TMs 
involves  using  SME  panels  to  achieve  similar  grouping.  While  there  is  merit  to  this  in  terms  of 
expediency,  it  opens  the  subsequent  analysis  to  suspicion  of  bias,  especially  if  the  SMEs  have  insufficient 
breadth  of  knowledge  of  and  experience  with  the  AFS.  And  the  foundation  of  the  TIDES  analysis  rests 
on  the  ability  of  other  SMEs  to  use  the  task  modules  to  recognize  jobs,  estimate  proportions  of  resources 
dedicated  to  training  for  them  (TT,  TRR  and  TRA),  and  to  estimate  both  formal  and  on-the-job  training 
required  to  achieve  proficiency  (LCs). 

Obviously,  a  clear  Job  Structure  is  essential  to  constructing  a  relevant  J&TH  survey.  If  a  particular  OAR 
is  conducted  without  a  reasonably  reliable  job  structure  analysis,  it  may  be  necessary  for  HRDD  to 
supplement  the  original  OAR  before  a  J&TH  survey  and  other  data  collection  instruments  can  be 
constructed. 

Training  Efficiency  and  Effectiveness  Methodology  (TEEM)  Requirements  -. 

The  TEEM  is  a  computer-based  method  for  integrating  data  from  occupational  analysis,  field  unit  inputs 
and  course  documents  to  identify  trends  of  under-  and  over-trained  tasks  for  the  puipose  of  improving 
formal  course  content  (Metrica,  2000;  Teachout,  Sego  &  Ford,  1995;  Teachout,  Sego,  &  Olea,  1993). 

The  conceptual  foundation  of  TEEM  is  that  in  technical  training,  tasks  with  high  Task  Learning  Difficulty 
(TLD)  should  receive  greater  emphasis  in  formal  training  than  those  with  lower  ratings.  TEEM  generates 
a  Cartesian  graph  (quadrant  1,  only)  reflecting  the  relative  position  of  tasks  trained  in  formal  courses  (see 
Figure  1,  below).  Along  the  X-axis,  tasks  are  rated  in  terms  of  time  dedicated  to  training  the  task  in  the 
course.  The  Y-axis  reflects  the  degree  of  task  learning  difficulty  the  task  received  in  AFOMS’  OAR.  The 
resultant  scatter  plot  can  be  analyzed,  focusing  on  those  tasks  outside  a  reasonable  boundary  of,  say,  one 
standard  deviation  from  the  regression  line,  and  judge  whether  the  excessive  time  is  spent  on  tasks 
relative  to  their  TLD  or  whether  time  balance  is  disproportionately  low. 
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Using  TEEM,  Training  Developers  and  Training  Managers  can  zero  in  on  those  tasks  which  appear  to  have 
the  greatest  disparity  between  degree  of  difficulty  and  amounts  of  time  dedicated  in  formal  courses.  The 
result  of  such  analyses  yields  opportunities  to  further  refine  and  improve  the  efficiency  and  effectiveness  of 
the  Air  Force’s  training  investment  in  each  specialty  where  it  is  employed. 

Simulation  Utility  Analysis  System  (SUMS): 

SUMS  was  developed  to  create  an  entity-based  personnel  flow  model  to  analyze  a  multi-job  system 
linking  job  performance  and  enlistment  standards  (Stone,  Turner,  Engquist,  &  Looper,  1993;  Stone, 
Turner,  Fast,  Curry,  Looper,  &  Engquist,  1992).  It  provides  capability  to  affect  such  human  capital 
management  factors  as  enlistment  standards,  job  classification  standards,  promotion  policies,  and  force¬ 
restructuring  initiatives.  The  SUMS  provides  managers  with  the  capability  to  analyze  aptitude  standards 
as  they  affect  force  structures,  experience  mixes,  and  force  productivity  at  the  career  field  level  and 
various  other  levels  of  aggregation.  In  addition,  the  system  can  evaluate  alternative  aptitude  and 
experience  mixes  within  and  across  AFSs. 

SUMS  operates  at  a  higher  level  of  abstraction  than  do  TIDES  or  TEEM,  however,  it  creates  models 
similar  to  TIDES  in  terms  of  establishing  a  baseline  of  aptitude  indicator  data  and  several  other  human 
capital  policies,  such  as  desired  retention  rates,  personnel  flow  through  various  stages  of  a  career,  and 
attrition  rates  as  cohorts  reach  career  milestones.  As  such,  it  provides  potential  to  supplement  or 
complement  projects  using  other  tools.  That  potential  is  enhanced  when  there  are  consistent  practices  in 
data  gathering  from  one  study  to  the  next. 

Acquisition  Information  Retrieval  and  Simulation  ( ACQUIRES )  System. 

The  ACQUIRES  system  is  a  tool  developed  specifically  to  model  human  capital  management  practices 
for  the  Air  Force  Systems  Command  Acquisition  Professional  Development  Program  (APDP). 
ACQUIRES  simulates  the  civilian  force  at  the  individual  entity  level  and  provides  extensive  database 
facilities  for  analyzing  and  summarizing  this  individual  level  data.  ACQUIRES  allows  for  the  analysis  of 
the  effects  of  APDP  policies  on  the  flow  and  content  of  the  civilian  acquisition  force  within  the  context  of 
a  simulation  and  projection  model.  Policymakers  could  analyze  short-term  and  long-term  implications  of 
changes  in  professional  education  requirements  on  the  civilian  force  and  the  future  course  needs  to  meet 
these  changes.  It  bears  passing  resemblance  to  TIDES  in  that  it  models  the  flow  of  a  particular  population 
through  a  regimen  of  highly  structured  training  and  experience  events  in  order  to  impose  discipline  on  a 
very  sensitive  Air  Force  issue,  the  handling  of  research  and  acquisition  funds. 
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As  a  template,  ACQUIRES  could  be  modified  to  address  other  sensitive  Air  Force  issues  (e.g.,  the  career 
management  of  those  charged  with  controlling  nuclear  weapons  materials,  or  defining  development 
patterns  of  highly  sensitive  and  visible  human  capital  assets  such  as  training  of  pilots  or  UAV  operators). 
In  such  cases,  availability  of  AFOMS  data  to  form  a  foundation  for  timely,  high-fidelity  simulation  of 
personnel  and  manpower  policies  and  practices  would  be  invaluable.  Key  data  from  AFOMS  would 
involve  job  structures  of  the  occupations,  differences  in  work  performed  by  progressively  more 
experienced  cohorts,  identification  of  key  competencies  (as  revealed  by  Task  Modules),  task  factor  data 
that  may  mitigate  or  validate  task  performance  data  with  respect  to  given  policies  and  background 
information  on  previously  held  jobs,  capacity  of  courses  to  support  volumes  of  trainees,  and  histories  of 
progressive  assignments  building  on  prior  experience. 

Statistical  Neural  Network  Application  Package  ( SNNAP )  : 

Statistical  Neural  Network  Application  Package  (SNNAP)  was  developed  to  be  able  to  analyze  artificial 
neural  network  models  of  decisions,  time-series  phenomenon,  system  control,  and  other  input-output 
relationships  (Wiggins,  Grobman,  and  Fooper,  1993;  Wiggins  and  Fooper,  1993).  SNNAP  implements 
three  network  architectures  (back  propagation,  probabilistic  neural  networks,  and  learning  vector 
quantization)  and  utilizes  special  training  heuristics  to  improve  out-of-sample  performance.  The  package 
was  developed  to  optimize  performance  on  highly  stochastic  databases  with  potentially  nonlinear  and 
interacting  features.  SNNAP  can  be  used  in  a  number  of  decision-support  systems,  and  while  AFOMS 
data  are  not  directly  needed  by  SNNAP,  those  data  can  be  used  in  many  modeling  and  simulation 
concepts  where  SNNAP  is  an  integral  feature. 

Job  Structuring  Technology  (JST)  Requirements-. 

JST  was  developed  as  a  way  of  providing  a  timely  decision  aid  to  assist  Air  Force  stakeholders  in 
evaluating  alternative  scenarios  for  achieving  a  large  body  of  work  over  a  career  life  cycle  (20  to  30 
years)  (Weissmuller,  Driskill,  Mitchell  and  Gosc,  1996).  JST  establishes  scenarios,  and  exemplar  entities 
representing  airmen  in  the  AFS  are  "run"  through  the  model  on  monthly  intervals,  where  exemplars  are 
promoted  and  reassigned.  Each  exemplar  carries  a  Proficiency  Folder  (PF)  which  records  module-level 
experience  &  proficiency  for  every  module  being  simulated.  This  PF  is  automatically  updated  to  reflect 
increased  proficiency  when  the  exemplar  attends  a  course  or  performs  on-the-job.  When  an  exemplar  is 
assigned  to  a  job,  the  proficiency  scores  from  the  PF  are  matched  against  specified  Job  Requirements  (JR) 
and  the  importance  of  those  requirements  to  the  mission.  The  weighted  percentage  of  job  incumbents 
exceeding  and  failing  minimum  standards  are  reported  for  each  month  as  a  goodness-of-fit  which  can  be 
reported  for  the  full  force,  within  a  specified  AFS,  or  for  a  given  task  module  in  the  master  folder. 
Running  a  JST  model  gives  stakeholders  a  chance  to  evaluate  the  effects  of  modifying  variables 
representing  human  capital  management  policies. 

The  key  data  needed  for  JST  operation  that  could  originate  from  AFOMS  data  includes: 

1.  Task  &  Duty  Fist  (from  USAF  Job  Inventory) 

2.  Percent  Members  Performing  (PMP)  for  selected  groups: 

Full  Group  (Everyone  in  the  AFSC) 

1-24  Months  Total  Active  Federal  Military  Service  (TAFMS)  (School/First  Assignment) 
25-48  Months  TAFMS  (Productive  Half  1st  Term) 

49-96  Months  TAFMS  (Second  Term) 

7  Skill  Fevels  (High  Tech/Management) 

3.  TD  -  Task  Fearning  Difficulty:  relative,  within  AFSC 

4.  BFD-IT  -  Benchmarked  Beaming  Difficulty  -  Individual  Task  level 

5.  Task  Co-Performance  Modules 

6.  Task  Co-Performance  Clustering  Solution  (CFUSEXT) 

7.  STS,  POI,  CTS,  CEC  documents  with  mappings  into  the  task  list 

8.  Enhanced  Task  Modules  (to  include  Equipment,  Knowledge,  etc.) 
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Phase  III.  In  analyzing  the  gap  between  current  AFOMS/OA  outputs  and  those  requirements  of  the 
HRRD  tools,  many  of  the  needed  data  elements  are  available;  however,  some  data  types  have  been 
discontinued  or  modified.  Four  important  gaps  were  noted  in  developing  this  plan;  limited  availability  of 
job  structure  information  in  some,  but  not  all,  OARs,  an  absence  of  co-performance  task  modules,  limited 
information  on  degree  of  inter-rater  reliability  on  task  factor  data,  and  uncertainty  about  being  able  to 
reconstruct  data  in  the  event  additional  analyses  are  required  to  be  able  to  support  AF/A1PF  in  responding 
to  Air  Force  stakeholders  regarding  such  issues  as  rationally  restructuring  the  force,  adjusting  recruiting 
standards  and  accession  goals,  or  modeling  potential  new  occupations. 

The  absence  of  thorough  job  structure  analyses,  based  on  hierarchical  clustering  of  cases  using  the  Ward 
Method  (Levine,  2003)  in  every  OAR  raises  the  specter  that  if  TIDES  or  TEEM  or  JST  analyses  should 
be  desired,  the  HRRD  may  be  faced  with  the  option  of  accepting  the  less-reliable  approach  of  using  a 
structure  solution  based  on  incumbent -selected  job  titles  or  on  performing  an  expensive  follow-up  job 
structure  analysis  using  the  traditional  hierarchical  clustering  method.  Running  such  a  follow-up  analysis 
would  be  predicated  on  being  able  to  reassemble  the  records  in  a  suitable  format  (probable,  but  not 
guaranteed)  from  the  current  ORACLE  output  format. 

The  absence  of  task  co-performance  clustering  could  place  an  additional  burden  on  any  TIDES  analysis 
or  JST  analysis.  That  burden  is  similar  to  the  one  created  by  an  absence  of  case  clustering;  the  ORACLE 
output  require  reformatting  for  tools  that  can  run  the  task  clustering  solution  using  Ward’s  Method. 

From  the  period  of  their  earliest  conceptions,  task  factor  data  were  routinely  collected  on  all  tasks  within  a 
given  OA  project.  In  an  effort  to  be  responsive  to  AFOMS’  primary  customers,  the  first  enlistment 
training  establishment  of  Air  Education  and  Training  Command,  the  strategy  for  collecting  Task  Learning 
Difficulty  in  most  surveys  has  been  restricted  primarily  to  technical  tasks,  omitting  the  supervisory, 
planning  and  evaluating,  and  some  administrative  tasks.  The  TEEM  tool  is  somewhat  impacted  by  the 
reduced  Task  Learning  Difficulty  data  collection;  however,  the  impact  is  not  insurmountable,  and  may 
even  be  negligible,  since  the  TEEM  focus  is  on  those  tasks  associated  with  technical  aspects  of  the  job. 

The  HRRD  may  need  the  capacity  to  conduct  BOLD  technique  to  estimate  the  relative  TLD  across  AFSs 
to  supporting  such  analyses  as  researching  appropriate  cut  scores  for  different  Air  Force  occupations  on 
the  Armed  Forces  Vocational  Aptitude  Battery.  Current  AFOMS  TLD  data  provide  valuable  source 
material  for  such  BOLD  studies,  but  there  are  still  a  few  unanswered  questions  about  AFOMS  task  factor 
data.  As  mentioned  earlier,  while  TE  and  TLD  values  were  present  in  the  samples  shown  to  this  author, 
the  information  about  their  inter-rater  reliability  was  not  evident. 

It  may  also  be  useful  for  AFPC/DPSF  to  be  able  to  resurvey  occupations  if  supplemental  data  are  needed 
to  fill  gaps  in  the  information  provided  by  AFOMS’  primary  surveys. 
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CONCLUSIONS 


The  Transition  Plan  for  moving  AFOMS  data  to  the  FIRRD  is  still  in  flux  due  to  ongoing  reorganization 
efforts.  While  a  decision  was  reached  in  early  May  by  the  AFOMS  Commander  to  “give  all  the  data”  to 
FIRRD  at  the  conclusion  of  each  OAR,  questions  remain  as  to  what  data  will  be  sent  and  whether  it  will 
be  interpretable  without  substantial  reverse  engineering.  The  effort  to  perform  this  contract  did  uncover 
some  positive  information  regarding  Occupational  Analysis  in  the  Air  Force.  While  there  have  been 
discussions  of  major  changes  to  the  methodology,  the  core  of  OA  is  still  present  in  the  work  of  AFOMS. 
The  concerns  uncovered  in  this  effort;  however,  should  not  be  overlooked.  First,  the  Air  Force  should 
have  some  assurance  that  current  methods  are  as  good  as,  or  superior  to  legacy  methods.  To  that  end, 
AFPC/DPSF  should  seek  to  obtain  published  demonstrations  of  the  equivalence  of  current  methods  to 
traditional  methods.  In  the  absence  of  those  demonstrations,  the  FIRRD  should  consider  conducting 
benchmarking  studies  on  a  small  but  representative  sample  of  completed  OARs,  to  determine  the  degree 
of  confidence  that  can  be  placed  in  the  AFOMS-published  job  structures.  In  the  absence  of  published 
head-to-head  comparisons,  it  seems  reasonable  that  AFOMS  would  at  least  tell  AFPC/DPSF  which  SPSS 
clustering  solutions  are  being  used. 

Related  to  that  matter  is  the  issue  of  being  able  to  follow-up  on  OAR  projects  with  additional  analyses.  A 
willingness  of  AFOMS  to  pass  on  copies  of  the  raw  data,  prior  to  input  into  ORACLE,  would  give 
assurances  that  the  original  records  can  be  reconstructed  for  additional  analyses.  In  the  absence  of  the 
original  raw  data  records,  it  is  imperative  that  HRRD  receive  not  just  the  large  volumes  of  relational  data, 
but  the  task  list  and  other  elements  of  the  survey  instrument,  (e.g.,  background  questions,  equipment  lists, 
and  special  analysis  questions). 

As  a  general  practice,  HRRD  should  avoid  wherever  possible  the  use  of  SME  panels  in  lieu  of  researched 
and  documented  quantitative  methods  for  determining  job  structure  and  task  module  solutions.  In  some 
cases,  that  may  be  necessary,  but  whenever  possible,  if  the  job  structure  isn’t  present,  and  in  all  cases 
where  co-performance  task  clustering  hasn’t  been  done,  consideration  should  be  given  to  making  those 
steps  part  of  any  analysis  plan  when  employing  the  research  tools  mentioned  above.  Having  the  ability 
to  resurvey  occupations  following  AFOMS’  OAR  publications  would  enable  AFPC/DPSF  to  address 
some  data  shortfalls. 

With  regard  to  task  factor  data,  it  would  be  useful  to  know  the  inter-rater  reliabilities  of  the  data  sets 
AFOMS  publishes  in  their  OARs,  even  if  AFOMS  doesn’t  want  to  put  them  in  their  published  reports. 
Each  occupation  is  unique,  and  each  occupation’s  task  factor  data  may  be  more  or  less  reliable.  Use  of 
unreliable  data  could  lead  to  a  false  confidence  by  users  who  don’t  understand  the  limitations  of  the  data, 
and  whose  subsequent  decisions  may  be  based  on  unsound  information. 
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